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Training  Collaboration  in  a  Network-assisted  Environment 


Introduction 

Previous  research  on  non-face-to-face  collaboration  in  network-enabled 
environments,  documented  in  Performance  in  a  distal  collaborative  environment 
(Schaab,  Dressel,  Sabol,  &  Rittman,  2007),  led  researchers  to  question  why  some 
participants  who  engaged  in  minimal  collaboration  reported  first,  that  collaboration  was 
necessary  to  succeed  and  second,  that  they  indeed  had  collaborated  during  the  game. 

If  participants  truly  thought  that  they  had  collaborated  when  they  did  not,  could  it  be  that 
they  did  not  know  how  to  collaborate  effectively?  Subsequent  observation  in  an 
experiment  comprised  of  a  large  group  of  former  military  personnel  suggested  a  similar 
lack  of  collaboration.  One  retired  Major  General  emphasized  a  need  to  “break  the 
stovepipes”  and  collaborate  with  others  to  get  information  to  those  who  needed  it  in  a 
timely  fashion.  Additionally,  he  noted  that  this  failure  to  collaborate  contributed  to 
reduced  situational  awareness.  In  order  to  determine  if  training  directed  on  how  to 
collaborate  would  enhance  information  sharing  and  improve  shared  situational 
awareness,  the  experiment  from  Schaab  et  al.  (2007)  was  modified  by  having  half  of  the 
participants  exposed  to  a  short  training  video  on  how  to  collaborate.  Those  who  did  not 
receive  the  training  on  how  to  collaborate  were  provided  with  an  equal  amount  of  time  to 
explore  the  game.  A  description  of  the  experiment  follows  along  with  a  brief  summary 
of  the  findings  from  the  initial  experiment  (see  Schaab  et  al.,  2007  for  detailed  findings). 
Findings  from  the  current  research  clearly  demonstrated  that  collaboration  training 
significantly  improved  performance. 

Research  concepts 

Research  Venue 

SCUDHunt,  an  on-line  game  developed  by  Thoughtlink,  INC  (www.thouqhtlink. 
com),  was  selected  for  this  research  on  collaboration  because  it  provided  a  simplified 
model  of  the  interplay  of  shared  awareness  and  collaboration,  while  permitting 
independent  manipulation  of  variables  thought  to  affect  them.  SCUDHunt  required 
participants  to  (1)  collaborate  from  distributed  locations  and  (2)  share  unique 
information  from  their  intelligence  assets  for  optimal  game  performance.  The  goal  of  the 
game  is  simple:  locate  three  SCUD  missile  launchers  on  a  map.  The  game  requires 
geographically  dispersed  players  to  collaborate  while  executing  digital  tasks  in  order  to 
achieve  a  shared  goal. 

Cognitive  Task  Analysis 

A  cognitive  task  analysis  of  SCUDHunt  identified  critical  points  where  collaboration 
would  be  beneficial  (Ross,  2003).  In  general,  players  needed  to  communicate  planning 
strategies  and  to  share  gathered  information  in  order  to  perform  effectively.  The 
collaboration  areas  identified  were: 


1 


1 .  SEARCH-  Generate  own  plan  and  coordinate  plan  with  partner  who  controls  other 
assets  for  asset  placement  in  order  to  gain  the  most  coverage  and  avoid  limitations; 

2.  DISCUSS  RESULTS-  Coordinate  to  find  out  what  assets  were  used  by  partner  to  get 
the  results,  consider  the  reliability  of  these  assets,  and  consider  any  previous  data  and 
interpretations;  and 

3.  GENERATE/COORDINATE  STRIKE  PLAN-  Coordinate  strike  plan  (where 
participants  think  SCUDs  are  located)  with  partner  to  verify  its  strength. 

Common  Knowledge 

This  situation  of  networked  individuals  who  have  shared  goals  but  unique  roles 
and  responsibilities  raised  the  question  of  collaboration  at  a  distance:  to  what  extent 
does  knowledge  about  a  partner’s  role  (all  condition)  influence  an  individual’s 
performance  effectiveness?  Findings  from  the  initial  experiment  (Schaab  et  al.,  2007) 
revealed  that  during  the  first  of  the  two  games  played,  no  differences  in  performance 
were  seen  between  those  trained  in  the  all  condition  and  those  trained  only  on  their  own 
assets  and  responsibilities  (own  condition)  in  either  quality  score  (number  of  SCUD 
launchers  located)  or  shared-situational  awareness  (number  of  identified  locations  in 
common).  In  the  second  game,  participants  in  the  all  condition  located  significantly  more 
SCUD  launchers  than  did  those  in  the  own  condition,  but  those  in  the  own  condition  had 
significantly  higher  levels  of  shared-situational  awareness.  This  means  that  participants 
who  were  cross-trained  in  both  their  role  and  in  their  partner’s  role  were  more 
successful  in  locating  SCUD  launchers.  Participants  trained  solely  in  their  role  achieved 
higher  levels  of  agreement  with  their  partner  on  where  they  thought  that  the  SCUD 
launchers  were  located,  but  were  wrong  more  frequently  than  those  who  were  cross 
trained. 

Communication  Mode 

Communication  mode  was  manipulated.  All  pairs  wore  headsets  that  allowed  oral 
communication  during  one  of  the  two  games  played;  during  the  other  game,  participants 
communicated  by  sending  typed  messages  via  an  on-screen  "chat"  box.  For  a  random 
half  of  the  pairs,  the  "chat"  game  came  first.  The  data  analyzed  included  measures  of 
the  types  and  frequency  of  communication  between  participants  in  the  "chat"  game. 
Players  with  a  higher  frequency  of  communications  in  the  categories  of  game  situation, 
player  status,  and  non-task  related/social,  identified  a  significantly  greater  number  of 
SCUDs.  No  significant  difference  in  quality  score  or  shared-situational  awareness  was 
found  during  the  initial  experiment  as  a  function  of  communication  mode,  therefore  text 
chat  was  used  for  all  communication  during  this  replication.  Data  on  the  number  of 
messages  sent  and  the  total  number  of  words  sent  were  collected  (Schaab  et  al.,  2007). 
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Method 


Participants 

Sixty-four  undergraduate  students,  28  females  and  36  males,  received  course 
credit  for  three  hours  of  participation.  Participants  ranged  in  age  from  18  to  45  years, 
with  the  majority,  75  percent,  between  18  to  25  years.  Four  participants  had  military 
experience. 

Materials 

Questionnaire.  At  the  beginning  of  the  session,  participants  completed  a 
questionnaire  requesting  demographic  information  (e.g.,  gender,  age,  and  military 
experience)  and  computer  experience. 

Workload.  The  NASA  Task  Load  Index  (TLX)  (Hart  &  Staveland,  1988)  is  a 
multi-dimensional  rating  tool  that  provides  workload  scores  based  on  six  subscales: 
Mental  Demands,  Physical  Demands,  Temporal  Demands,  Own  Performance,  Effort, 
and  Frustration.  The  NASA  TLX  was  administered  to  each  participant  after  completing 
each  of  two  games  of  SCUDHunt. 

Game  Description.  SCUDHunt  is  a  representation  of  a  situation  in  which 
Soldiers  would  use  digital  systems  to  execute  tasks  requiring  collaboration.  The 
SCUDHunt  game  presented  players  with  the  mission  of  determining  where  -  on  a  five- 
by-five  grid  board  representing  the  map  of  a  hostile  country  -  the  launchers  for  SCUD 
missiles  were  located.  Participants  were  told  that  there  were  three  SCUD  launchers, 
each  in  a  different  fixed  location  among  the  25  squares  on  the  board.  On  each  of  five 
turns,  participants  deployed  intelligence-gathering  assets  (for  example,  a 
reconnaissance  satellite  or  a  team  of  Navy  Seals),  received  reports  from  those  assets, 
and  created  a  “strike  plan”  (to  be  sent  to  their  fictional  commander)  indicating  their  best 
guess  based  on  all  the  information  received  both  from  that  turn  and  previous  turns  as  to 
the  SCUD  launcher  locations.  They  were  told  that  the  final  strike  plan  -  after  the  fifth 
turn  -  would  be  used  by  their  commander  to  direct  an  attack  on  the  SCUD  launchers, 
and  they  were  given  the  results  of  this  final  strike  plan  in  terms  of  which  bombed 
location  held  a  now-destroyed  launcher.  Participants  controlled  either  air  or  ground 
assets  (see  Figure  1)  with  each  asset  having  unique  capabilities  and  returning 
intelligence  reports  of  different  reliabilities.  For  example,  eyes  on  target  reports  from 
human  intelligence  assets  would  be  more  reliable  than  reports  generated  from  satellites 
where  sensors  which  must  interpret  images  from  great  distances. 
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Air  Control 


Ground  Control 


Manned  Aircraft 


HUMINT  SEAL 


Figure  1.  Participants  controlled  either  air  assets  or  ground  assets. 

Measures.  Generated  as  the  game  was  played  were  1)  the  number  of  SCUD 
launcher  locations  correctly  identified,  called  quality  score,  2)  the  degree  to  which  the 
two  participants  on  a  team  chose  the  same  grid  squares  or  location  in  their  independent 
strike  plans,  called  shared-situational  awareness,  3)  the  number  of  text  chat 
communications  taking  place,  4)  measures  of  subjective  workload  reported  on  the 
NASA  TLX,  and  5)  responses  to  questionnaire  items  on  demographics  and  computer 
experiences. 

Design.  The  primary  independent  variable  for  this  experiment  was  training  on 
how  to  collaborate,  with  half  of  the  participants  randomly  assigned  to  receive  this 
training.  Variable  manipulations  were  identical  to  the  initial  experiment.  Common 
knowledge  on  all  versus  own  training  conditions  involved  training  on  the  characteristics 
of  the  information-gathering  assets  used  in  the  SCUDHunt  game  (see  Figure  2).  Every 
participant  received,  as  their  first  training  module,  an  explanation  of  the  characteristics 
of  the  assets  they  would  be  controlling.  Half  of  the  pairs  (the  own  condition)  received  a 
second  exposure  to  the  same  asset  training;  the  other  half  (the  all  condition)  received 
training  in  which  each  participant  learned  the  characteristics  of  assets  to  be  controlled 
by  that  participant's  partner.  Half  of  the  participants  in  the  all  condition  and  half  in  the 
own  condition  received  a  short  training  segment  on  how  to  collaborate  which  was 
presented  via  computer  to  ensure  consistency.  Those  who  did  not  receive  this  training 
were  allowed  an  equal  amount  of  time  to  practice  the  game.  Training  consisted  of 
fourteen  Power  Point  slides  with  voice  over.  Participants  were  asked  a  question 
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followed  by  the  answer  on  the  subsequent  slide.  For  example,  a  view  of  the  response 
grid  was  shown  with  results  from  the  initial  query.  Participants  in  the  training  condition 
were  asked,  “Is  this  a  good  initial  plan?  Why?”  The  following  slide  described  why  it  was 
a  good  plan  (initial  search  covered  a  wide  area  with  no  duplication). 


All 

Own 

Collaboration 

Training 

16 

16 

No 

Collaboration 

Training 

16 

16 

Figure  2.  Design. 

Procedure.  Upon  arrival  at  the  laboratory,  individual  participants  were  sent  to 
separate  rooms  where  they  read  and  signed  a  standard  consent  form  describing  the 
experiment  and  their  rights  as  participants.  This  was  followed  by  each  participant 
completing  a  questionnaire  on  demographics  and  their  experience  with  computers  and 
computer  games.  Researchers  then  explained  that  the  experiment  involved  participants 
playing  a  computer  game  with  a  partner  who  was  located  in  another  room. 

Several  computer-based  training  modules  were  then  presented  on  1)  the  overall 
aspects  of  playing  the  SCUDHunt  game  and  2)  the  characteristics  of  the  information¬ 
gathering  assets  used  in  playing  the  game.  Participants  took  paper  and  pencil  quizzes 
on  the  material  just  presented  following  each  training  module  and  were  given  immediate 
corrective  feedback,  if  necessary,  to  ensure  that  they  understood  how  to  play  the  game 
and  the  capabilities  of  their  assets.  Half  of  the  participants  were  provided  with  a  short 
training  module  on  how  to  collaborate.  The  remaining  half  was  given  an  equal  amount 
of  time  to  practice  the  game.  After  this  training,  the  pair  played  a  one-turn  practice 
game,  to  ensure  that  the  mechanics  of  playing  the  game  were  understood.  After  the 
experimenters  answered  any  question  the  participants  might  have,  the  pair  played  two 
complete  five-turn  games  of  SCUDHunt.  During  these  games,  data  were  automatically 
collected  on  1)  the  messages  participants  sent  to  each  other,  2)  the  degree  to  which 
grid  squares  chosen  as  targets  in  the  "strike  plans"  (submitted  at  the  end  of  each  turn) 
were  identical  for  the  two  members  of  the  pair,  and  3)  the  number  of  those  chosen 
target  squares  that  actually  contained  missile  launchers. 

Results 

Quality  Scores.  Participants  who  received  training  on  how  to  collaborate  located 
more  SCUDs,  thereby  receiving  higher  quality  scores,  during  each  of  the  five  turns  of 
both  games  (see  Figure  3).  Results  from  Analyses  of  Variance  (ANOVAs)  for  each 
game  show  that  those  receiving  collaboration  training  received  significantly  higher 
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quality  scores  than  those  who  did  not  receive  the  training.  ANOVA  results  for  game  1 
were  F  (1 , 62)  =  6.373,  p  <.05,  eta  squared  =  .093,  and  results  for  game  2  were  F  (1 , 
61 )  =  7.084,  p  <.05,  eta  squared  =  .104 
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Figure  3.  Comparison  of  Quality  Scores  between  those  who  received  training  on 
collaboration  and  those  who  did  not  receive  the  training. 

Shared  Situational  Awareness.  Similarly,  participants  who  received  collaboration 
training  scored  higher  on  shared  situational  awareness  during  every  trial  (see  Figure  4). 
ANOVA  results  for  game  1  were  F(1 , 62)  =  14.442,  p  <.05,  eta  squared  =  .189  and  for 
game  2  were  F(1, 62)  =  11.759,  p<. 05,  eta  squared  =  .159. 

Workload.  No  significant  difference  in  workload  was  found  between  the  groups 
who  had  collaboration  training  and  those  who  did  not. 

All  and  Own  Training  Conditions.  No  significant  difference  was  found  between 
quality  scores  or  shared  situational  awareness  when  comparing  those  who  were  trained 
in  the  all  and  the  own  condition.  These  findings  differ  from  those  found  in  the  initial 
experiment.  It  is  possible  that  the  additional  time  provided  both  conditions 
(collaboration  training  or  additional  practice)  may  have  increased  participants  overall 
proficiency  in  playing  the  game,  thereby  reducing  the  influence  of  training  on  both  their 
own  and  their  partners  tasks. 
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Figure  4.  Comparison  of  Quality  Scores  between  those  who  received  training  on 
collaboration  and  those  who  did  not  receive  the  training.  No  locations  in  common=0;  All 
3  locations  in  common=2. 


Figure  5.  Comparison  of  number  of  words  exchanged  during  game  1  and  game  2  as  a 
function  of  whether  participants  received  collaboration  training. 
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Figure  6.  Comparison  of  number  of  messages  exchanged  during  game  1  and  game  2 
as  a  function  of  whether  participants  received  collaboration  training. 

Number  of  Words  and  Messages  Exchanged.  Those  who  received  collaboration 
training  exchanged  significantly  more  words  with  their  partner  (F(1 , 62)  =  5.36,  p  <  .05, 
eta  squared  =  .08)  and  exchanged  significantly  more  messages  (F(1 , 62)  =  4.75,  p  < 
.05,  eta  squared  =  .071)  than  those  who  did  not  receive  the  collaboration  training  (see 
Figures  5  and  6). 


Discussion 

Collaboration  is  a  commonly  used  but  ill-defined  term.  Technology  enables  more 
and  more  non-face-to-face  collaboration,  with  the  promise  of  enhanced  information 
sharing  and  shared-situational  awareness.  But  technology  alone  is  not  the  answer.  As 
technology  advances,  there  is  an  increased  need  to  understand  how  humans 
collaborate  at  a  distance.  Findings  from  this  research  advance  our  knowledge  of  how  to 
enhance  non-face-to-face  collaboration.  Our  initial  research  suggested  that  participants 
do  not  have  a  clear  understanding  of  how  to  collaborate  in  non-face-to-face  computer 
mediated  environments.  When  participants  were  provided  with  a  short  training  video 
that  asked  them  to  think  about  how  and  when  they  should  collaborate,  followed  by  a 
demonstration  of  collaboration  on  the  task,  their  performance  in  locating  SCUD  missiles 
and  in  their  shared-situational  awareness  improved  significantly.  Additionally,  partners 
engaged  in  more  frequent  communications.  These  findings  suggest  that  training  on 
digital  systems  should  include  a  segment  on  how  to  effectively  use  these  systems  to 
solicit,  share,  and  combine  information  collaboratively. 
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Slide  A2 


Collaboration 


I 

% 


People  actively,  sharing 
information,  knowledge, 
perceptions,  or  concepts 
when  working  together 
toward  a  common  gurgose. 


Collaboration  is  necessary  for  mission  success  in  SCUDHunt. 

The  Army  defines  collaboration  as  “People  actively  sharing  information,  knowledge, 
perceptions,  or  concepts  when  working  together  towards  a  common  purpose.”  Here  are  the 
areas  where  you  need  to  collaborate. 
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Slide  A3 


0 

Coordination 


Slide  A4 


Successful  collaboration  improves  your  chances  of  locating  the  3  Scud  missile  launchers  in 
SCUDHhunt.  Coordinate  with  your  ally  to  determine  how  to  deploy  your  assets  to  cover  the 
most  territory  in  your  searches. 
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Slide  A5 


Search  Plan 

•  Locate  3  SCUD  launchers  within  a  5  x  5  grid 


What  is  the  best  way  to  begin  your  search?  Delay  going  to  next  slide. 

Slide  A6 


Team  members  deployed  their  intelligence  assets  and  these  are  the  intelligence  reports  from  turn 
1.  You  can  see  that  the  team  coordinated  their  search  because  they  maximized  their  coverage  of 
the  search  area  on  this  first  search.  There  is  one  location  where  this  search  may  not  have  been 
well  coordinated.  Look  at  location  D4.  It  looks  like  2  intelligence  assets  searched  the  same  area. 
Because  this  is  the  first  search,  this  may  not  be  the  best  use  of  the  assets.  If  this  was  not  the  first 
search,  the  assets  at  D4  may  be  rechecking  a  suspected  area  for  the  Scud. 
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Slide  A7 


Search  Results  from 
Turn  1 

Do  the  results  indicate 
that  the  team 
coordinated  their 
search? 

Is  this  a  good  search 
plan? 

Why? 


Here  are  the  intelligence  reports  from  another  team  for  the  first  search. 

It  looks  like  these  team  members  did  not  coordinate  because  they  used  their  intelligence 
resources  to  search  the  same  areas.  Remember,  this  is  the  first  search  and  it  is  wise  to  gather  as 
much  information  as  quickly  as  possible.  In  subsequent  turns  you  may  want  to  take  a  second 
look  at  some  areas  based  on  the  intelligence  report  and  the  reliability  of  the  asset  reporting. 

Slide  A8 


Coordinate  your  Search 


For  example,  coordinate  your  search  with  your  ally  so  that  you  cover  as  much  of  the  map  area  as 
possible.  We  don’t  have  any  reports  from  column  B.  I’ll  search  that  area.  Humlnt-I’ll  stay  at  C3 
and  report  next  turn.  UAV-Is  there  an  area  that  has  not  been  searched? 
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Slide  A9 


After  deploying  your  assets,  you  receive  intelligence  reports  of  the  results.  Remember  that  you 
and  your  ally  control  different  assets  and  the  intelligence  from  these  assets  may  have  limited 
reliability.  You  need  to  share  the  information  on  the  reliability  of  your  assets  with  your  ally  to 
effectively  interpret  the  results. 

Slide  A10 


Satellite  Intelligence  Report 


Histoiy  of  Search  Resulls 


You  can  see  the  intelligence  report  from  each  of  your  assets  by  clicking  on  the  tab  of  that  asset 
on  the  left-hand  map.  The  results  appear  on  the  map  on  the  right.  You  will  see  the  results  one 
asset  at  a  time. 
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Slide  All 


The  shared  viz  view  shows  you  the  results  from  all  assets  deployed  during  this  turn.  These  are 
the  intelligence  reports  from  Turn  2.  What  does  it  mean?  Are  there  really  scuds  located  at  A2  and 
B2?  Collaborate  with  your  ally  to  determine  what  asset  reported  a  scud  sighting.  How  reliable 
are  the  reports  from  those  assets?  How  about  the  green  0  indicating  nothing  to  report?  How 
confident  are  you  that  there  really  is  nothing  at  those  locations?  Again,  check  with  your  ally  to 
determine  what  intelligence  provided  that  information  and  discuss  how  reliable  the  report  is 

Slide  A12 


Satellite  reports  nothing 
spotted  in  areas  A2-E2. 
Satellite  often  misses  a  Scud 
when  one  is  there. 


Air  Control 


Roger.  Special  operations 
reports  Scud  at  A2. 
SpecOps  reports  are  very 
reliable. 


Ground  Control 
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Slide  A13 


After  deploying  your  assets,  you  receive  intelligence  reports  of  the  results. 

Remember  that  you  and  your  ally  control  different  assets  and  the  intelligence  from  these  assets 
may  have  limited  reliability.  You  need  to  share  the  information  on  the  reliability  of  your  assets 
with  your  ally  to  meaningfully  interpret  the  search  results. 
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Slide  A14 


Strike  Plan 


Locate  3  SCUD  launchers  within  a  5  x  5  grid 


map: 


2  S  4  5 


A  7  ?  ?  ?  ? 


B 


?  ?  ?  ? 


C  ?  ?  ?  ?  ? 


D  ?  ?  ?  ?  ? 


E  ?  ?  ?  ?  ? 


You  have  planned  with  your  ally  on  where  to  deploy  your  combined  assets  and  received  the 
resulting  intelligence  reports  from  these  assets.  Now  you  must  develop  a  Strike  Plan  to  send  to 
your  commander.  This  Strike  Plan  tells  your  commander  the  most  likely  location  of  the  3  scuds 
based  on  the  intelligence  that  you  have  at  this  time.  Remember  that  the  scuds  do  not  move, 
therefore  combine  the  intelligence  reports  from  previous  searches  with  your  current  search 
results  to  arrive  at  your  best  guess.  Your  commander  does  NOT  want  you  to  discuss  the  3  areas 
that  you  recommend  striking  with  your  ally.  Your  commander  wants  separate  interpretations  of 
the  intelligence  reports  from  you  and  your  ally.  Therefore,  you  and  your  ally  will  develop  your 
own  Strike  Plan. 
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Slide  A15 


Search  Results 


Tum  Q1  Phase  Strike  Plan 


History  of  Search  Results 


smallest  numlkc  ol  *qu«*s 

Jari  Spec  Ops  |  COMNT  |  HUMNT  SM^Rar  |  ±_± 

rWaw|T-,3|T,„.|T«,6| 

Jontipet  Ops  |  COMNT  |  HUMNT  ShsedVc 

JorSSppcOp*|  COMINT  HUMINT  SlskpPWs  |  «l  » 

1  ZH 

_Tyn?!|  Twn3|  Tar*  |  TsjtsSI 

p*c  Ops  |  COMINT  |  MLNINT  SNowIVk  | 

Jliii 
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Here  are  the  search  results  from  Turn  1  and  Tum  2.  Remember  that  the  Scuds  do  not  move. 
Based  on  these  intelligence  reports,  where  what  Strike  Plan  would  you  recommend  to  your 
commander?  You  may  discuss  these  reports  with  your  ally.  For  example,  you  would  want  to 
know  how  reliable  the  reports  are  from  A2  and  B3  because  those  assets  reports  a  Scud  present. 
The  ?  Indicates  some  type  of  vehicle,  possibly  a  Scud.  How  likely  are  those  assets  to  report 
some  type  of  vehicle  if  there  is  a  Scud  at  that  location?  When  you  feel  that  you  understand  the 
intelligence  reports  and  have  some  ideas  on  where  the  Scuds  are  located,  you  and  your  ally,  on 
your  own,  select  the  3  most  likely  areas  where  the  Scuds  could  be  located. 
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Slide  A16 


Voices. 


Slide  A17 


Remember,  collaborate  with  your  ally,  share  what  you  know.  You  are  both  in  this  together.  Good 
luck  and  good  hunting! 
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